Introduction
This document describes the process for using the R/Exams package for creating an exam composed of simple single choice questions, the creation of an export file, and its uploading for import into Canvas.
Install Library and Dependencies
It is assumed that RStudio is already installed, if not please install it. With RStudio running do the following:
- In the library where the exam or exams are to be built for a given class, create a new RStudio project.
- Create a new R Markdown file. It might be called something like
Exam-Creation.Rmd. This file will be used for installing the library, copying the question template, building the exam, making temporary copies of the exam for proof reading, and finally generating the export file to upload to Canvas.
- In the newly created R Markdown file, add the following chunk to install the
exams package and load it. Note that once the package has been installed on a computer, this line can be removed or commented out.
install.packages("exams", dependencies = TRUE)
library(exams)
It is important to note that pandoc needs to be installed on the system. Unless there is a need to generate pdf output, everything should be ready to start building questions for an exam. See the R/Exams instillation tutorial for additional guidance in instillation issues.
Creating Questions
In the main R Markdown file, the one we’re calling Exam-Creation.Rmd create a new chunk and add the following code which can be used to copy the template file to create a new file in the same working directory, and open it for editing:
file.copy("question-template.Rmd", "question001.Rmd")
file.edit("question001.Rmd")
I suggest numbering the questions as “question001.Rmd, question002.Rmd, etc.” Doing this makes building an exam easier because the file pattern is simple. Just change the output file and file edit numbers for each new question to be created and run that chunk again.
As described below, it is possible to have Canvas randomize the order of answers when presenting the questions on a quiz. If randomizing the answer is desirable, avoid writing questions that rely on answers such as “all of the above” or “none of the above.” Instead, use wording like “all of the answers” or “none of the answers.”
Build an Exam and Check It
- Create a new code chunk using the example below.
- The code creates a variable
e that contains a vector consisting of all the R Markdown files to be included in the exam. Method 1 (commented out) illustrates how to build the vector literally. Method 2 (preferable) shows how to use pattern matching to combine all Rmd files in the directory with the name question.
- The code applies the function
exams2html() and exams2pandoc() to the vector e to create html and .doc output for review. The onboard help forexams2canvas() is pretty clear. To call help ?exams2canvas().
- The function
exams_metainfo(exams2pandoc(e)) is used to generate an exam key. This might be useful for proof reading and ensuring the answers are correct.
- Ctrl+Shift+Enter to run the script, or run it line by line to test it.
# Method 1: List the files literally
# e <- c("Question01.Rmd", "Question02.Rmd")
# Method 2: List the files based on extension
e <- list.files(pattern = "question.*.Rmd")
# Write file to html for a quick check to see everything is ok
exams2html(e)
# Write file to Word Doc just to have a copy all in one place.
exams2pandoc(e)
# Create an exam key for proofreading purposes.
# Copy the console output to a text file.
exams_metainfo(exams2pandoc(e))
Editing questions
- If either
exams2html() or exams2pandoc() do not run properly, provided pandoc is installed properly, the issue may be due to unwanted white spaces in the question (see above).
- When proof reading either an html or doc version of the exam and errors or corrections are found in a question or questions, it is possible to search through all of the *.Rmd files in the current working directory using the
Edit menu >Find in Files option. Paste in the string or search term to identify which of the question files needs to be edited. RStudio will show which of the questionXXX.Rmd files needs to be edited.
- Once the necessary edits are made to the individual
questionXXX.Rmd files and re-run the code chunk above to generate new output exams. Once the exam is in proper order, it is time to create an export file.
Create an Export File to Send to Canvas
To create an export file, apply the function exams2canvas(). This will generate a zipped QTI file which can be uploaded. At this stage, be sure to properly set how many points each question is worth. However, I think this can be adjusted in a Canvas question bank.
# Export the exam to a zip file for Canvas
exams2canvas(e, name = "tempquiz", points = 3)
When importing into canvas:
- Import the new questions into a temporary quiz.
- In Canvas, create a new quiz and from the quiz create a new question group.
- Link that question group to a question bank, set the correct number of questions and point value.
- It is possible to randomize the order of both the questions and the answers. If the plan is to randomize the answers.
LS0tDQp0aXRsZTogIk1ha2luZyBTaW1wbGUgTXVsdGlwbGUgQ2hvaWNlIEV4YW1zIHdpdGggUi1FeGFtcyBhbmQgSW1wb3J0aW5nIHRvIENhbnZhcyINCmF1dGhvcjogIk5hdGhhbiBDcmFpZyINCmRhdGU6ICI5IEF1Z3VzdCAyMDIwIg0Kb3V0cHV0Og0KICBodG1sX2RvY3VtZW50Og0KICAgIHRvYzogeWVzDQogICAgbnVtYmVyX3NlY3Rpb25zOiB5ZXMNCiAgICBsaW5rLWNpdGF0aW9uczogeWVzDQogICAgZGZfcHJpbnQ6IHBhZ2VkDQpiaWJsaW9ncmFwaHk6IFtyZWZlcmVuY2VzLmJpYiwgcGFja2FnZXMuYmliXQ0KLS0tDQoNCmBgYHtyLCBpbmNsdWRlPUZBTFNFfQ0KbGlicmFyeShEaWFncmFtbWVSKQ0KbGlicmFyeShleGFtcykNCmBgYA0KDQojIEludHJvZHVjdGlvbg0KDQpUaGlzIGRvY3VtZW50IGRlc2NyaWJlcyB0aGUgcHJvY2VzcyBmb3IgdXNpbmcgdGhlIGBSL0V4YW1zYCBwYWNrYWdlIGZvciBjcmVhdGluZyBhbiBleGFtIGNvbXBvc2VkIG9mIHNpbXBsZSBzaW5nbGUgY2hvaWNlIHF1ZXN0aW9ucywgdGhlIGNyZWF0aW9uIG9mIGFuIGV4cG9ydCBmaWxlLCBhbmQgaXRzIHVwbG9hZGluZyBmb3IgaW1wb3J0IGludG8gQ2FudmFzLg0KDQpgYGB7ciwgZWNobz1GQUxTRX0NCm1lcm1haWQoIg0KZ3JhcGggVEINCiAgQVtFeGFtLUNyZWF0aW9uLlJtZF0tLS1CW3F1ZXN0aW9uLXRlbXBsYXRlLlJtZF0NCiAgQi0tPkNbcXVlc3Rpb24wMDEuUm1kXQ0KICBCLS0+RFtxdWVzdGlvbjAwMi5SbWRdDQogIEItLT5FW3F1ZXN0aW9uMDAzLlJtZF0NCiAgQy0tPkZbRXhhbSBWZWN0b3JdDQogIEQtLT5GDQogIEUtLT5GDQogIEYtLT5HW0hUTUwgVmVyc2lvbl0NCiAgRi0tPkhbRG9jIFZlcnNpb25dDQogIEYtLT5JW0Fuc3dlciBLZXldDQogIEYtLT5KW1FUSSBaaXBdDQogIEotLT5LW1VwbG9hZCBRVEkgWmlwIEZpbGUgdG8gQ2FudmFzXQ0KIikNCmBgYA0KDQojIyBJbnN0YWxsIExpYnJhcnkgYW5kIERlcGVuZGVuY2llcw0KDQpJdCBpcyBhc3N1bWVkIHRoYXQgW1JTdHVkaW9dKGh0dHBzOi8vd3d3LnJzdHVkaW8uY29tLykgaXMgYWxyZWFkeSBpbnN0YWxsZWQsIGlmIG5vdCBwbGVhc2UgaW5zdGFsbCBpdC4gV2l0aCBSU3R1ZGlvIHJ1bm5pbmcgZG8gdGhlIGZvbGxvd2luZzoNCg0KMS4gIEluIHRoZSBsaWJyYXJ5IHdoZXJlIHRoZSBleGFtIG9yIGV4YW1zIGFyZSB0byBiZSBidWlsdCBmb3IgYSBnaXZlbiBjbGFzcywgY3JlYXRlIGEgbmV3IFJTdHVkaW8gcHJvamVjdC4NCjIuICBDcmVhdGUgYSBuZXcgUiBNYXJrZG93biBmaWxlLiBJdCBtaWdodCBiZSBjYWxsZWQgc29tZXRoaW5nIGxpa2UgYEV4YW0tQ3JlYXRpb24uUm1kYC4gVGhpcyBmaWxlIHdpbGwgYmUgdXNlZCBmb3IgaW5zdGFsbGluZyB0aGUgbGlicmFyeSwgY29weWluZyB0aGUgcXVlc3Rpb24gdGVtcGxhdGUsIGJ1aWxkaW5nIHRoZSBleGFtLCBtYWtpbmcgdGVtcG9yYXJ5IGNvcGllcyBvZiB0aGUgZXhhbSBmb3IgcHJvb2YgcmVhZGluZywgYW5kIGZpbmFsbHkgZ2VuZXJhdGluZyB0aGUgZXhwb3J0IGZpbGUgdG8gdXBsb2FkIHRvIENhbnZhcy4NCjMuICBJbiB0aGUgbmV3bHkgY3JlYXRlZCBSIE1hcmtkb3duIGZpbGUsIGFkZCB0aGUgZm9sbG93aW5nIGNodW5rIHRvIGluc3RhbGwgdGhlIGBleGFtc2AgcGFja2FnZSBhbmQgbG9hZCBpdC4gTm90ZSB0aGF0IG9uY2UgdGhlIHBhY2thZ2UgaGFzIGJlZW4gaW5zdGFsbGVkIG9uIGEgY29tcHV0ZXIsIHRoaXMgbGluZSBjYW4gYmUgcmVtb3ZlZCBvciBjb21tZW50ZWQgb3V0Lg0KDQpgYGB7ciwgZXZhbCA9IEZBTFNFfQ0KaW5zdGFsbC5wYWNrYWdlcygiZXhhbXMiLCBkZXBlbmRlbmNpZXMgPSBUUlVFKQ0KbGlicmFyeShleGFtcykNCmBgYA0KDQpJdCBpcyBpbXBvcnRhbnQgdG8gbm90ZSB0aGF0IFtwYW5kb2NdKGh0dHBzOi8vcGFuZG9jLm9yZy8pIG5lZWRzIHRvIGJlIGluc3RhbGxlZCBvbiB0aGUgc3lzdGVtLiBVbmxlc3MgdGhlcmUgaXMgYSBuZWVkIHRvIGdlbmVyYXRlIHBkZiBvdXRwdXQsIGV2ZXJ5dGhpbmcgc2hvdWxkIGJlIHJlYWR5IHRvIHN0YXJ0IGJ1aWxkaW5nIHF1ZXN0aW9ucyBmb3IgYW4gZXhhbS4gU2VlIHRoZSBSL0V4YW1zIFtpbnN0aWxsYXRpb24gdHV0b3JpYWxdKGh0dHA6Ly93d3cuci1leGFtcy5vcmcvdHV0b3JpYWxzL2luc3RhbGxhdGlvbi8pIGZvciBhZGRpdGlvbmFsIGd1aWRhbmNlIGluIGluc3RpbGxhdGlvbiBpc3N1ZXMuDQoNCiMjIFF1ZXN0aW9uIEZvcm1hdA0KDQpXaGlsZSBhIHNpbmdsZSBmaWxlIGlzIHVzZWQgdG8gbG9hZCB0aGUgbGlicmFyeSBhbmQgZ2VuZXJhdGUgdGhlIGV4YW0sIGVhY2ggaW5kaXZpZHVhbCBleGFtIHF1ZXN0aW9uIGlzIHdyaXR0ZW4gdG8gaXRzIG93biBzZXBhcmF0ZSBSIE1hcmtkb3duIGAuUm1kYCBmaWxlLiBUaGVzZSBmaWxlcyBhcmUgbGF0ZXIgY29tYmluZWQgdG9nZXRoZXIgdG8gYnVpbGQgdGhlIGV4YW0uIEl0IGlzIGltcG9ydGFudCB0aGF0IGVhY2ggZXhhbSBxdWVzdGlvbiBmaWxlIGlzIHByb3Blcmx5IHdyaXR0ZW4uIFRvIGZhY2lsaXRhdGUgY29uc2lzdGVuY3kgaW4gZmlsZSBzdHJ1Y3R1cmUsIGFuZCB0byBzYXZlIHRpbWUgaXQgaXMgcmVjb21tZW5kZWQgdG8gdXNlIGEgdGVtcGxhdGUgZmlsZS4gVGhlIGZvbGxvd2luZyB0ZW1wbGF0ZSBpcyBmb3Igc2luZ2xlIGNob2ljZSBxdWVzdGlvbnMuIE90aGVyIHRlbXBsYXRlcyBhcmUgYXZhaWxhYmxlIGluIHRoZSBSL0V4YW1zIFtGaXJzdCBTdGVwc10oaHR0cDovL3d3dy5yLWV4YW1zLm9yZy90dXRvcmlhbHMvZmlyc3Rfc3RlcHMvKSB0dXRvcmlhbC4NCg0KICAgIFF1ZXN0aW9uDQogICAgPT09PT09PT0NCiAgICBXaGljaCBhbnN3ZXIgaXMgY29ycmVjdA0KDQogICAgQW5zd2VybGlzdA0KICAgIC0tLS0tLS0tLS0NCiAgICAqIEFuc3dlciAxDQogICAgKiBBbnN3ZXIgMg0KICAgICogQW5zd2VyIDMNCiAgICAqIEFuc3dlciA0DQoNCiAgICBNZXRhLWluZm9ybWF0aW9uDQogICAgPT09PT09PT09PT09PT09PQ0KICAgIGV4bmFtZTogcXVlc3Rpb24yDQogICAgZXh0eXBlOiBzY2hvaWNlDQogICAgZXhzb2x1dGlvbjogMDAxMA0KDQpBIGZldyBwaWVjZXMgb2YgYE1ldGEtaW5mb3JtYXRpb25gIGFuZCBnZW5lcmFsIGlzc3VlcyB0byBjb25zaWRlciBhcmUgd29ydGggbWVudGlvbmluZyB3aGVuIHdyaXRpbmcgYW5kIGVkaXRpbmcgcXVlc3Rpb25zOg0KDQotICAgYGV4bmFtZWAgaXMgdGhlIHF1ZXN0aW9uIG5hbWUuDQotICAgYGV4dHlwZWAgaXMgdGhlIHF1ZXN0aW9uIHR5cGUuIEZvciBzaW1wbGUgc2luZ2xlIGNob2ljZSBxdWVzdGlvbnMsIGBzY2hvaWNlYCBpcyB0aGUgb3B0aW9uIHRvIHNlbGVjdC4NCi0gICBgZXhzb2x1dGlvbmAgaXMgdGhlIGFuc3dlciBrZXkuIEVhY2ggYW5zd2VyIG11c3QgYmUgbWFya2VkIGFzIGNvcnJlY3QgYDFgIG9yIGluY29ycmVjdCBgMGAuIEluIHRoZSBzYW1wbGUgZmlsZSBhYm92ZSwgQW5zd2VyIDMgaXMgdGhlIGNvcnJlY3QgYW5zd2VyLg0KLSAgIE1ha2Ugc3VyZSB0aGF0IHRoZXJlIGlzIG5vIGJsYW5rIGxpbmUgYmV0d2VlbiB0aGUgYD09PT1gIG9yIHRoZSBgLS0tYCBhbmQgdGhlIHRleHQgb2YgdGhlIHF1ZXN0aW9uLCBvciBhbnN3ZXIgbGlzdC4NCi0gICBUaGVyZSBzaG91bGQgYmUgb25seSBhIHNpbmdsZSBibGFuayBsaW5lIGFmdGVyIHRoZSBxdWVzdGlvbiBzdGVtIGFuZCBhZnRlciB0aGUgYW5zd2VyIGxpc3QuDQotICAgU3BhY2VzIGF0IHRoZSBlbmQgb2YgYW4gYW5zd2VyIGNhbiBjYXVzZSBwcm9ibGVtcyB3aGVuIGJ1aWxkaW5nIGEgdGVzdC4gSWYgdGhlcmUgYXJlIHRyYWlsaW5nIHdoaXRlIHNwYWNlcywgdGhlIHF1ZXN0aW9ucyB3aWxsIG5vdCBjb21waWxlIHByb3Blcmx5IGludG8gYW4gZXhhbS4gR29pbmcgYmFjayBhbmQgZmluZGluZyB0aGVzZSBpc3N1ZXMgY2FuIGJlIHRpbWUgY29uc3VtaW5nIGFuZCBmcnVzdHJhdGluZy4gUHJldmVudGluZyB0aGVtIGluIHRoZSBmaXJzdCBwbGFjZSBoZWxwcyB0cmVtZW5kb3VzbHkuICoqUHJvYmFibHkgdGhlIGJlc3Qgd2F5IHRvIHByZXZlbnQgYW5kIGZpbmQgdW53YW50ZWQgd2hpdGUgc3BhY2VzIG9yIGNhcnJpYWdlIHJldHVybnMgaXMgYnkgZW5hYmxpbmcgd2hpdGUgc3BhY2UgZGlzcGxheSBpbiBSU3R1ZGlvKiouIEVuYWJsZSB3aGl0ZSBzcGFjZSBkaXNwbGF5IGJ5IGdvaW5nIHRvOiBUb29scyBcPiBHbG9iYWwgT3B0aW9ucyBcPiBDb2RlIFw+IERpc3BsYXkgXD4gdGljayB0aGUgc2VsZWN0aW9uIGJveCB0byBlbmFibGUgIlNob3cgd2hpdGUgc3BhY2UgY2hhcmFjdGVycyIgXD4gYW5kIGNsaWNrICJPSyIuDQoNCiMgQ3JlYXRpbmcgUXVlc3Rpb25zDQoNCkluIHRoZSBtYWluIFIgTWFya2Rvd24gZmlsZSwgdGhlIG9uZSB3ZSdyZSBjYWxsaW5nIGBFeGFtLUNyZWF0aW9uLlJtZGAgY3JlYXRlIGEgbmV3IGNodW5rIGFuZCBhZGQgdGhlIGZvbGxvd2luZyBjb2RlIHdoaWNoIGNhbiBiZSB1c2VkIHRvIGNvcHkgdGhlIHRlbXBsYXRlIGZpbGUgdG8gY3JlYXRlIGEgbmV3IGZpbGUgaW4gdGhlIHNhbWUgd29ya2luZyBkaXJlY3RvcnksIGFuZCBvcGVuIGl0IGZvciBlZGl0aW5nOg0KDQpgYGB7ciwgZXZhbD1GQUxTRX0NCiAgICBmaWxlLmNvcHkoInF1ZXN0aW9uLXRlbXBsYXRlLlJtZCIsICJxdWVzdGlvbjAwMS5SbWQiKQ0KICAgIGZpbGUuZWRpdCgicXVlc3Rpb24wMDEuUm1kIikNCmBgYA0KDQpJIHN1Z2dlc3QgbnVtYmVyaW5nIHRoZSBxdWVzdGlvbnMgYXMgInF1ZXN0aW9uMDAxLlJtZCwgcXVlc3Rpb24wMDIuUm1kLCBldGMuIiBEb2luZyB0aGlzIG1ha2VzIGJ1aWxkaW5nIGFuIGV4YW0gZWFzaWVyIGJlY2F1c2UgdGhlIGZpbGUgcGF0dGVybiBpcyBzaW1wbGUuIEp1c3QgY2hhbmdlIHRoZSBvdXRwdXQgZmlsZSBhbmQgZmlsZSBlZGl0IG51bWJlcnMgZm9yIGVhY2ggbmV3IHF1ZXN0aW9uIHRvIGJlIGNyZWF0ZWQgYW5kIHJ1biB0aGF0IGNodW5rIGFnYWluLg0KDQpBcyBkZXNjcmliZWQgYmVsb3csIGl0IGlzIHBvc3NpYmxlIHRvIGhhdmUgQ2FudmFzIHJhbmRvbWl6ZSB0aGUgb3JkZXIgb2YgYW5zd2VycyB3aGVuIHByZXNlbnRpbmcgdGhlIHF1ZXN0aW9ucyBvbiBhIHF1aXouIElmIHJhbmRvbWl6aW5nIHRoZSBhbnN3ZXIgaXMgZGVzaXJhYmxlLCBhdm9pZCB3cml0aW5nIHF1ZXN0aW9ucyB0aGF0IHJlbHkgb24gYW5zd2VycyBzdWNoIGFzICJhbGwgb2YgdGhlIGFib3ZlIiBvciAibm9uZSBvZiB0aGUgYWJvdmUuIiBJbnN0ZWFkLCB1c2Ugd29yZGluZyBsaWtlICJhbGwgb2YgdGhlIGFuc3dlcnMiIG9yICJub25lIG9mIHRoZSBhbnN3ZXJzLiINCg0KIyBCdWlsZCBhbiBFeGFtIGFuZCBDaGVjayBJdA0KDQoxLiAgQ3JlYXRlIGEgbmV3IGNvZGUgY2h1bmsgdXNpbmcgdGhlIGV4YW1wbGUgYmVsb3cuDQoyLiAgVGhlIGNvZGUgY3JlYXRlcyBhIHZhcmlhYmxlIGBlYCB0aGF0IGNvbnRhaW5zIGEgdmVjdG9yIGNvbnNpc3Rpbmcgb2YgYWxsIHRoZSBSIE1hcmtkb3duIGZpbGVzIHRvIGJlIGluY2x1ZGVkIGluIHRoZSBleGFtLiBNZXRob2QgMSAoY29tbWVudGVkIG91dCkgaWxsdXN0cmF0ZXMgaG93IHRvIGJ1aWxkIHRoZSB2ZWN0b3IgbGl0ZXJhbGx5LiBNZXRob2QgMiAocHJlZmVyYWJsZSkgc2hvd3MgaG93IHRvIHVzZSBwYXR0ZXJuIG1hdGNoaW5nIHRvIGNvbWJpbmUgYWxsIFJtZCBmaWxlcyBpbiB0aGUgZGlyZWN0b3J5IHdpdGggdGhlIG5hbWUgYHF1ZXN0aW9uYC4NCjMuICBUaGUgY29kZSBhcHBsaWVzIHRoZSBmdW5jdGlvbiBgZXhhbXMyaHRtbCgpYCBhbmQgYGV4YW1zMnBhbmRvYygpYCB0byB0aGUgdmVjdG9yIGBlYCB0byBjcmVhdGUgaHRtbCBhbmQgLmRvYyBvdXRwdXQgZm9yIHJldmlldy4gVGhlIG9uYm9hcmQgaGVscCBmb3JgZXhhbXMyY2FudmFzKClgIGlzIHByZXR0eSBjbGVhci4gVG8gY2FsbCBoZWxwIGA/ZXhhbXMyY2FudmFzKClgLg0KNC4gIFRoZSBmdW5jdGlvbiBgZXhhbXNfbWV0YWluZm8oZXhhbXMycGFuZG9jKGUpKWAgaXMgdXNlZCB0byBnZW5lcmF0ZSBhbiBleGFtIGtleS4gVGhpcyBtaWdodCBiZSB1c2VmdWwgZm9yIHByb29mIHJlYWRpbmcgYW5kIGVuc3VyaW5nIHRoZSBhbnN3ZXJzIGFyZSBjb3JyZWN0Lg0KNS4gIEN0cmwrU2hpZnQrRW50ZXIgdG8gcnVuIHRoZSBzY3JpcHQsIG9yIHJ1biBpdCBsaW5lIGJ5IGxpbmUgdG8gdGVzdCBpdC4NCg0KYGBge3IsIGV2YWw9RkFMU0V9DQogICAgIyBNZXRob2QgMTogTGlzdCB0aGUgZmlsZXMgbGl0ZXJhbGx5DQogICAgIyBlIDwtIGMoIlF1ZXN0aW9uMDEuUm1kIiwgIlF1ZXN0aW9uMDIuUm1kIikNCg0KICAgICMgTWV0aG9kIDI6IExpc3QgdGhlIGZpbGVzIGJhc2VkIG9uIGV4dGVuc2lvbg0KICAgIGUgIDwtIGxpc3QuZmlsZXMocGF0dGVybiA9ICJxdWVzdGlvbi4qLlJtZCIpDQoNCiAgICAjIFdyaXRlIGZpbGUgdG8gaHRtbCBmb3IgYSBxdWljayBjaGVjayB0byBzZWUgZXZlcnl0aGluZyBpcyBvaw0KICAgIGV4YW1zMmh0bWwoZSkNCg0KICAgICMgV3JpdGUgZmlsZSB0byBXb3JkIERvYyBqdXN0IHRvIGhhdmUgYSBjb3B5IGFsbCBpbiBvbmUgcGxhY2UuDQogICAgZXhhbXMycGFuZG9jKGUpDQoNCiAgICAjIENyZWF0ZSBhbiBleGFtIGtleSBmb3IgcHJvb2ZyZWFkaW5nIHB1cnBvc2VzLg0KICAgICMgQ29weSB0aGUgY29uc29sZSBvdXRwdXQgdG8gYSB0ZXh0IGZpbGUuDQogICAgZXhhbXNfbWV0YWluZm8oZXhhbXMycGFuZG9jKGUpKQ0KYGBgDQoNCiMgRWRpdGluZyBxdWVzdGlvbnMNCg0KLSAgIElmIGVpdGhlciBgZXhhbXMyaHRtbCgpYCBvciBgZXhhbXMycGFuZG9jKClgIGRvIG5vdCBydW4gcHJvcGVybHksIHByb3ZpZGVkIHBhbmRvYyBpcyBpbnN0YWxsZWQgcHJvcGVybHksIHRoZSBpc3N1ZSBtYXkgYmUgZHVlIHRvIHVud2FudGVkIHdoaXRlIHNwYWNlcyBpbiB0aGUgcXVlc3Rpb24gKHNlZSBhYm92ZSkuDQotICAgV2hlbiBwcm9vZiByZWFkaW5nIGVpdGhlciBhbiBodG1sIG9yIGRvYyB2ZXJzaW9uIG9mIHRoZSBleGFtIGFuZCBlcnJvcnMgb3IgY29ycmVjdGlvbnMgYXJlIGZvdW5kIGluIGEgcXVlc3Rpb24gb3IgcXVlc3Rpb25zLCBpdCBpcyBwb3NzaWJsZSB0byBzZWFyY2ggdGhyb3VnaCBhbGwgb2YgdGhlIFwqLlJtZCBmaWxlcyBpbiB0aGUgY3VycmVudCB3b3JraW5nIGRpcmVjdG9yeSB1c2luZyB0aGUgYEVkaXRgIG1lbnUgXD5gRmluZCBpbiBGaWxlc2Agb3B0aW9uLiBQYXN0ZSBpbiB0aGUgc3RyaW5nIG9yIHNlYXJjaCB0ZXJtIHRvIGlkZW50aWZ5IHdoaWNoIG9mIHRoZSBxdWVzdGlvbiBmaWxlcyBuZWVkcyB0byBiZSBlZGl0ZWQuIFJTdHVkaW8gd2lsbCBzaG93IHdoaWNoIG9mIHRoZSBgcXVlc3Rpb25YWFguUm1kYCBmaWxlcyBuZWVkcyB0byBiZSBlZGl0ZWQuDQotICAgT25jZSB0aGUgbmVjZXNzYXJ5IGVkaXRzIGFyZSBtYWRlIHRvIHRoZSBpbmRpdmlkdWFsIGBxdWVzdGlvblhYWC5SbWRgIGZpbGVzIGFuZCByZS1ydW4gdGhlIGNvZGUgY2h1bmsgYWJvdmUgdG8gZ2VuZXJhdGUgbmV3IG91dHB1dCBleGFtcy4gT25jZSB0aGUgZXhhbSBpcyBpbiBwcm9wZXIgb3JkZXIsIGl0IGlzIHRpbWUgdG8gY3JlYXRlIGFuIGV4cG9ydCBmaWxlLg0KDQojIENyZWF0ZSBhbiBFeHBvcnQgRmlsZSB0byBTZW5kIHRvIENhbnZhcw0KDQpUbyBjcmVhdGUgYW4gZXhwb3J0IGZpbGUsIGFwcGx5IHRoZSBmdW5jdGlvbiBgZXhhbXMyY2FudmFzKClgLiBUaGlzIHdpbGwgZ2VuZXJhdGUgYSB6aXBwZWQgUVRJIGZpbGUgd2hpY2ggY2FuIGJlIHVwbG9hZGVkLiBBdCB0aGlzIHN0YWdlLCBiZSBzdXJlIHRvIHByb3Blcmx5IHNldCBob3cgbWFueSBwb2ludHMgZWFjaCBxdWVzdGlvbiBpcyB3b3J0aC4gSG93ZXZlciwgSSB0aGluayB0aGlzIGNhbiBiZSBhZGp1c3RlZCBpbiBhIENhbnZhcyBxdWVzdGlvbiBiYW5rLg0KDQpgYGB7ciwgZXZhbD1GQUxTRX0NCiAgICAjIEV4cG9ydCB0aGUgZXhhbSB0byBhIHppcCBmaWxlIGZvciBDYW52YXMNCiAgICBleGFtczJjYW52YXMoZSwgbmFtZSA9ICJ0ZW1wcXVpeiIsIHBvaW50cyA9IDMpDQpgYGANCg0KV2hlbiBpbXBvcnRpbmcgaW50byBjYW52YXM6DQoNCi0gICBJbXBvcnQgdGhlIG5ldyBxdWVzdGlvbnMgaW50byBhIHRlbXBvcmFyeSBxdWl6Lg0KLSAgIEluIENhbnZhcywgY3JlYXRlIGEgbmV3IHF1aXogYW5kIGZyb20gdGhlIHF1aXogY3JlYXRlIGEgbmV3IHF1ZXN0aW9uIGdyb3VwLg0KLSAgIExpbmsgdGhhdCBxdWVzdGlvbiBncm91cCB0byBhIHF1ZXN0aW9uIGJhbmssIHNldCB0aGUgY29ycmVjdCBudW1iZXIgb2YgcXVlc3Rpb25zIGFuZCBwb2ludCB2YWx1ZS4NCi0gICBJdCBpcyBwb3NzaWJsZSB0byByYW5kb21pemUgdGhlIG9yZGVyIG9mIGJvdGggdGhlIHF1ZXN0aW9ucyBhbmQgdGhlIGFuc3dlcnMuIElmIHRoZSBwbGFuIGlzIHRvIHJhbmRvbWl6ZSB0aGUgYW5zd2Vycy4NCg==